Automatic detection of red lesions in digital color fundus photographs

ABSTRACT

Disclosed is an automated method which can detect images containing red lesions. First, each image is preprocessed; next, candidate objects that may represent red-lesions are extracted; and in the final stage the probability for each candidate to represent a red-lesion is estimated using a classifier and a large set of specifically designed features.

CROSS REFERENCE TO RELATED PATENT APPLICATION

This application claims priority to U.S. Provisional Application No.60/666,868 filed Mar. 31, 2005, herein incorporated by reference in itsentirety.

BACKGROUND

Diabetic retinopathy (DR) is the leading cause of blindness in theworking population of the western world. It is an eye disease which insome form afflicts 17% of diabetic subjects five years after diagnosisof diabetes and 97% fifteen years after diagnosis. Early diagnosisthrough regular screening and timely treatment has been shown to preventvisual loss and blindness. Digital color fundus photography allowsacquisition of fundus images (see 500 in FIG. 5) in a non-invasivemanner which is a prerequisite for large scale screening.

In a DR screening program, the number of fundus images that need to beexamined by ophthalmologists can be prohibitively large. The number ofimages without any sign of DR in a screening setting is typically over90%. Therefore an automated system that can decide whether or not anysigns suspicious for DR are present in an image can improve efficiency;only those images deemed suspect by the system would require examinationby an ophthalmologist.

Signs of DR include red lesions such as microaneurysms and intraretinalhemorrhages, and white lesions, such as exudates and cottonwool spots.Red lesions are among the first unequivocal signs of DR. Therefore,their detection is critical for a pre-screening system.

Previously published methods for the detection of red lesions havefocused on detecting microaneurysms in fluorescein angiography images ofthe fundus. In this type of image, the contrast between themicroaneurysms and background is larger than in digital color images.However, the mortality of 1:222,000 associated with the intravenous useof fluorescein prohibits the application of this technique forlarge-scale screening purposes.

The detection method described by Spencer, Cree, Frame and co-workersemployed a mathematical morphology technique that eliminates thevasculature from a fundus image yet left possible microaneurysmcandidates untouched. Spencer, J. Olson, K. McHardy, P. Sharp, and J.Forrester, “An image-processing strategy for the segmentation andquantification in fluorescein angiograms of the ocular fundus,”Computers and biomedical research, vol. 29, pp. 284-302, 1996. A. Frame,P. Undrill, M. Cree, J. Olson, K. McHardy, P. Sharp, and J. Forrester,“A comparison of computer based classification methods applied to thedetection of microaneurysms in ophthalmic fluorescein angiograms,”Computers in Biology and Medicine, vol. 28, pp. 225-238, 1998, allincorporated by reference in their entireties. It has also been applied,in a modified version, to high resolution red-free fundus images with asensitivity of 85% and a specificity of 76% on a per image basis.

A number of other approaches for the detection of red lesions in colorfundus images have also been described. A neural network has been usedto detect both hemorrhages and exudates. Each image was divided into20×20 pixel grids; these were then individually classified. The perimage results showed a sensitivity of 88.4% and a specificity of 83.5%.

A recursive region-growing procedure has been applied to segment boththe vessels and red lesions in a fundus image. Next, a neural networkwas used to detect the vessels exclusively. The objects that remainafter removal of the detected vasculature are labeled as microaneurysms.The evaluation was carried out on 10×10 pixel grids and not forindividual images or lesions. A sensitivity of 77.5% and specificity of88.7% were reported.

Results of a commercially available automatic red lesion detectionsystem have been released. However, their method was not described. Thesystem had a sensitivity of 93.1% and specificity of 71.4% on a perpatient basis. Because of the inadequacy of the aforementionedtechniques, there exists a great need for a screening system that canidentify lesions suspicious for Diabetic Retinopathy.

SUMMARY OF INVENTION

Provided is an automated method which can detect images containing redlesions with a very high sensitivity and a reasonable specificity.First, each image is preprocessed; next, candidate objects that mayrepresent red lesions are extracted; and in the final stage, theprobability for each candidate to represent a red lesion is estimatedusing a classifier and a large set of specifically designed features.

Other systems, methods, and aspects and advantages of the invention willbe discussed with reference to the figures and to the detaileddescription of the preferred embodiments.

BRIEF DESCRIPTION OF THE FIGURES

The invention will be described by way of example, in the description ofexemplary embodiments, with particular reference to the accompanyingfigures:

FIG. 1 shows an exemplary flow diagram outlining the general steps ofthe method.

FIG. 2 shows an exemplary flow diagram outlining the steps of the ImagePre-Processing step.

FIG. 3 a shows an exemplary flow diagram outlining the steps of theMathematical Morphology Based Candidate Detection Step.

FIG. 3 b shows an exemplary flow diagram outlining the steps of thePixel Classification Based Candidate Detection Step.

FIG. 3 c shows an exemplary flow diagram outlining the steps of thehybrid Candidate Detection Step.

FIG. 4 shows an exemplary flow diagram outlining the CandidateClassification step.

FIG. 5 shows a digital color fundus photograph containing red lesions.

FIG. 6 a shows part of the green color plane of a fundus image. Shownare segments of vasculature and several red lesions. The bright lesionscalled exudates are also a sign of diabetic retinopathy. Circles markthe location of some of the red lesions in the image.

FIG. 6 b shows a fundus image after subtracting a median filteredversion of the green plane large background gradients are removed.

FIG. 6 c shows a fundus image where all pixels with a positive value areset to zero to eliminate the bright lesions in the image. Note that theexudates often partially occlude red lesions. The non-occluded parts ofthe red lesions show up clearly in this image. An example of this ismarked with a rectangle.

FIG. 6 d shows the pixel classification result produced by the contrastenhancement step. The non-occluded parts of the hemorrhages are visibletogether with the vasculature and a number of red lesions.

FIG. 7 a shows the thresholded probability map.

FIG. 7 b shows the remaining objects after connected component analysisand removal of the large vasculature. The same red lesions as in FIG. 6a are indicated with a circle.

FIG. 7 c shows the shape and size of the extracted objects in FIG. 7 bdoes not correspond well with the actual shape and size of the objectsin the original image. A final region growing procedure is used to growback the actual objects in the original image which are shown here. Thesame red lesions as in FIG. 6 a are indicated with a circle.

FIG. 8 a shows ROC curves of the hybrid system (HS), the mathematicalmorphology based system using the original 13 features used in theSpencer-Frame System (MMO) and the human expert (HE). The sensitivityand specificity are on a per image basis.

FIG. 8 b shows FROC curves of the HS, MMO and HE. The sensitivity of thedetection is given on a per lesion basis. The average number of falsepositives per image is taken over all images in the test set. Thehorizontal axis has a logarithmic scale.

FIG. 9 shows an example of a final result of the hybrid red lesiondetection system. All detected objects have been marked on the imageusing different shapes. When an object belongs to multiple classes (e.g.is both a true positive (TP), of the hybrid system and a false negative(FN) of the human expert) the indicator of the second class is displayedsmaller close to the first class.

FIG. 10 shows an example of a computer based system implementing thedisclosed method.

DETAILED DESCRIPTION

Before the present systems and methods are disclosed and described, itis to be understood that the terminology used herein is for the purposeof describing particular embodiments only and is not intended to belimiting. While described below for use in detecting red lesions, thesystem and method for automatic detection of red lesions in digitalcolor fundus images can be used for detection of other objectsindicative of disease including, but not limited to, microaneurysms, dothemorrhages, flame-shaped hemorrhages, sub-intimal hemorrhages,sub-retinal hemorrhages, pre-retinal hemorrhages, micro-infarctions,cotton-wool spots, exudates and drusen.

Definitions

As used in the specification and the appended claims, the singular forms“a,” “an” and “the” include plural referents unless the context clearlydictates otherwise. Thus, for example, reference to “a lesion” includesmixtures of two or more such lesions, and the like.

Ranges can be expressed herein as from “about” one particular value,and/or to “about” another particular value. When such a range isexpressed, another embodiment includes from the one particular valueand/or to the other particular value. Similarly, when values areexpressed as approximations, by use of the antecedent “about,” it willbe understood that the particular value forms another embodiment. Itwill be further understood that the endpoints of each of the ranges aresignificant both in relation to the other endpoint, and independently ofthe other endpoint. It is also understood that there are a number ofvalues disclosed herein, and that each value is also herein disclosed as“about” that particular value in addition to the value itself. Forexample, if the value “10” is disclosed, then “about 10” is alsodisclosed. It is also understood that when a value is disclosed that“less than or equal to” the value, “greater than or equal to the value”and possible ranges between values are also disclosed, as appropriatelyunderstood by the skilled artisan. For example, if the value “10” isdisclosed the “less than or equal to 10” as well as “greater than orequal to 10” is also disclosed. It is also understood that thethroughout the application, data are provided in a number of differentformats, and that these data, represent endpoints and starting points,and ranges for any combination of the data points. For example, if aparticular data point “10” and a particular data point 15 are disclosed,it is understood that greater than, greater than or equal to, less than,less than or equal to, and equal to 10 and 15 are considered disclosedas well as between 10 and 15.

In this specification and in the claims which follow, reference will bemade to a number of terms which shall be defined to have the followingmeanings:

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

An “image” is defined as a reproduction or imitation of the form of aperson or thing, or specific characteristics thereof, in digital form.An image can be, but is not limited to, a JPEG image, a PNG image, a GIFimage, a TIFF image, or any other digital image format known in the art.“Image” is used interchangeably with “photograph.”

An “object” is defined as a tangible entity with defined borderscontained within a digital photograph. An object can be a pixel or agroup of pixels.

A “feature(s)” is defined as a group of one or more descriptivecharacteristics of objects that can discriminate for disease. A featurecan be a numeric attribute.

A “set of features” is defined as a customized group of one or moredescriptive characteristics of objects which most accurately classifyobjects indicative of eye disease.

A “threshold” is defined as a level, point, or value above whichsomething is true or will take place and below which it is not or willnot, such levels, points, or values include probabilities, sizes inpixels, and values representing pixel brightness.

“Thresholding” is defined as modifying pixels that contain acharacteristic either above or below a selected threshold value.

An “abnormal object” is defined as an object in a retinal imagerepresenting a finding of disease, including, but not limited to,microaneurysms, dot hemorrhages, flame-shaped hemorrhages, sub-intimalhemorrhages, sub-retinal hemorrhages, pre-retinal hemorrhages,micro-infarctions, cotton-wool spots, and yellow exudates.

A “supervised procedure” is defined as a computer programming methodwherein a program learns a function from examples of inputs and outputs.

A “seed coordinate” is defined as a location in an image that serves asa starting point for a region-growing procedure.

A “classifier” is defined as comprising mapping from unlabeled objectsto labeled objects. Classifiers can have a form (for example, decisiontree) plus an interpretation procedure (including how to handleunknowns, etc.). Some classifiers also provide probability estimates,which can be thresholded to yield an object decision thereby taking intoaccount a utility function.

“Boosting” is defined as a general method for training an ensemble ofclassifiers with a view to improving performance relative to that of asingle classifier.

“Support vector machines” are defined as learning machines that canperform binary classification (pattern recognition) and real valuedfunction approximation (regression estimation) tasks. Support VectorMachines non-linearly map their n-dimensional input space into a highdimensional feature space. In this high dimensional feature space alinear classifier is constructed.

A “Spencer-Frame” system/method is defined as using a limitedmathematical morphology based object detection system with features 1-13from the feature list disclosed herein.

“Image_(ORG)” is defined as any image initially input into the disclosedmethod.

“Image_(SC)” is defined as any image that has been shade-corrected.

“Image_(BG)” is defined as a background image.

“Vasculature map” is defined as an image containing the disposition orarrangement of blood vessels in an organ or other part of an organism.

“Image_(LESION)” is defined as an image containing mainly non-elongatedstructures.

“Image_(MATCH)” is defined as an image that had a filter applied.

“Image_(BIN-MM)” is defined as a binary image resulting fromMathematical Morphology Based Candidate Object Detection.

“Image_(PP)” is defined as any image that has had bright lesionsremoved.

“Image_(PROB)” is defined as any image where the pixels in the imagehave been assigned a probability.

“Image_(BIN-PC)” is defined as a binary image resulting from PixelClassification Based Candidate Object Detection.

“Mathematical morphology based candidate object detection” is defined ascomprising at least one of the following steps:

a. applying a morphological transformation of a first image todiscriminate circular from elongated regions producing a vasculaturemap;

b. subtracting the vasculature map from the first image to produce asecond image;

c. applying a matched filter to the second image to enhance the contrastbetween background regions and abnormal regions producing a third image;

d. applying a brightness threshold to the third image to produce afourth image; and

e. applying a region-growing procedure to the first image to generate aset of candidate objects.

“Supervised pixel classification based candidate object detection” isdefined as comprising at least one of the following steps:

a. training a classifier with a supervised procedure using examplepixels extracted from a labeled reference standard training set;

b. obtaining pixel feature vectors from a first image comprisingapplying filters for determining pixel intensities or othercharacteristics of pixels at each pixel location or pixel region in thefirst image;

c. automatically classifying each of a plurality of the first imagepixels as foreground or background pixels;

d. assigning a posterior probability of being a foreground pixel to eachfirst image pixel according to the equation p=n/k, producing a secondimage;

e. removing the second image pixels where the posterior probability isless than a probability threshold, producing a third image;

f. removing any object in the third image with a size that is above asize threshold;

g. applying a region-growing procedure to the first image to generate aset of candidate objects; and

h. filling empty pixels within the grown objects of the set of candidateobjects.

Throughout this application, various publications are referenced. Thedisclosures of these publications in their entireties are herebyincorporated by reference into this application in order to more fullydescribe the state of the art to which this pertains. The referencesdisclosed are also individually and specifically incorporated byreference herein for the material contained in them that is discussed inthe sentence in which the reference is relied upon.

Method

I. Image Preprocessing

To prepare fundus images for red lesion extraction, preprocessing isperformed on the green-plane of the original RGB color image,Image_(ORG). As red lesions have the highest contrast with thebackground in the green color plane information from the red and bluecolor planes are not used in this step.

a. Shade Correction

Fundus images often contain an intensity variation in the backgroundacross the image, called “vignetting” (see FIG. 5). This effect candeteriorate the performance of the candidate selection system. Any slowgradients in the background of Image_(ORG) can be removed, resulting ina “shade corrected” image Image_(SC). This can be accomplished byestimating the background image Image_(BG) and subtracting that fromImage_(ORG). Image_(BG) can be produced by median filtering Image_(ORG)with a 25×25 pixel kernel. The size of the median filter can be chosensuch that it is wider than the widest blood vessel in the dataset ofimages. A typical result of the operation is shown in FIG. 6 b.

b. Bright Lesion Removal

Bright pathologies can also appear in retinas affected by DR. Theselesions can have well defined sharp edges. When they lie close together,small “islands” of normal retina are formed between them. These can bepicked up as false positives (FPs) by the candidate object detection inthe pixel classification based system in later steps. To prevent this,these bright pathologies can be removed as follows.

The shade corrected image Image_(SC) has negative values for all pixelswhich have an intensity lower than the background. By removing allpixels with a positive value from the image, bright pathologies nolonger influence the later analysis. All pixels in Image_(SC) with apositive value can be set to zero resulting in a preprocessed imageImage_(PP). FIG. 6 c shows a fundus image where all pixels with apositive value are set to zero to eliminate the bright lesions in theimage. Note that the exudates often partially occlude red lesions. Thenon-occluded parts of the red lesions 602 show up clearly in this image.An example of this is marked with a rectangle.

II. Candidate Detection

Next, objects that are potential red lesions are detected in thepre-processed images. These objects can be called candidate objects.

a. Mathematical Morphology Based Candidate Detection

After the image pre-processing stage, the actual candidate objects canbe extracted from Image_(SC). To discriminate between circular,non-connected red lesions and the elongated vasculature, a morphologicaltop-hat transformation can be used. The method is not limited to the useof the top-hat transformation; other morphological transformations whichtogether achieve the same result as the morphological top-hattransformation can be used. This operation is based on morphologicallyopening the image with a linear structuring element at differentorientations. Twelve rotated structuring elements with a length of 9pixels and width of 1 pixel can be used with a radial resolution of 15°.The method is not limited to the use of twelve rotated structuringelements or a length of 9 pixels and a width of 1 pixel. The method isalso not limited to a radial resolution of 15°. As the resolution of animage increases the optimal length and radial resolution can increase.The length of the structuring element should be such that it is largerthan the biggest red lesion present in the set. A length of 9 pixelsgave the best balance between vessel segmentation and red lesiondetection for the set. In each of the 12 opened images, only those partsof the vasculature in which the linear structuring element can fitremain. By taking the maximum pixel value at each pixel location in all12 images, a map of only the vasculature can be obtained. Thisvasculature map can be subtracted from Image_(SC), resulting in animage, Image_(LESION), containing mainly non-elongated structures suchas, but not limited to, red lesions.

To enhance the contrast between background and red lesions inImage_(LESION), a matched filter can be used. The matched filter is a 2DGaussian with σ at least 1 pixel. The method is not limited to such amatched filter or such a scale; any filter that achieves similar resultscan be used. Next the filtered image, Image_(MATCH), can be thresholdedto produce a binary image Image_(BIN-MM). The threshold can be fixed ata certain level above the modal value of the image Image_(MATCH). Inthis implementation, 4 was used as the brightness threshold. This methodis not limited to the use of 4 as the brightness threshold. Thebrightness threshold can comprise values from 2 to 6. For example,brightness threshold values can be 2, 3, 4, 5, and 6. The extractedbinary objects are not a good representation of the pathologies as foundin Image_(ORG). A region growing procedure can be used to grow back theoriginal pathologies, in the Image_(MATCH). The darkest pixel under eachof the binary objects can serve as a starting point.

The estimated background image Image_(BG) obtained in the preprocessingstep can be used to find the threshold t for the region growingprocedure as follows:t=i _(seed) −x(i _(seed) −i _(bg)),  (1)where i_(seed) is the intensity at the starting position, i.e., thepixel under the binary object with the lowest gray value. Then i_(bg) isthe intensity of the same pixel in the background image and Xε[0, 1].Here x=0.5. Growing can start in the seed pixel and stop when no moreconnected pixels below the threshold can be found. The grown objectstogether can form a candidate object set. This method is not limited tothe use of this type of region growing procedure; alternativesegmentation methods that achieve similar results can be used.

b. Pixel Classification Based Candidate Detection

A limitation of the previously described candidate extraction techniqueis that any red lesions which are larger than the linear structuringelement cannot be detected. When the length of the structuring elementis increased to be able to detect larger objects, the vesselsegmentation deteriorates leading to more spurious candidate objectsbeing detected on the vasculature. This limitation can be removed byusing a pixel classification based method. This approach can detectlarger candidate objects and reduces the number of spurious candidateobjects on the vessels by integrating a vessel segmentation techniquewith red lesion detection. Both vasculature and possible red lesions canbe extracted at once, and subsequently the vasculature can be separatedfrom the red lesion candidate objects.

The digital color images used have a relatively low resolution, suchthat small red lesions typically have areas as low as four pixels, withrelatively low contrast. Such is the image quality of data currentlyused routinely for DR screening. With pixel classification the contrastbetween foreground pixels (red lesions and vasculature) and pixels inthe background can be improved such that a global threshold can be usedto extract all relevant objects in the image.

Pixel classification is a supervised method, so it needs to be trainedusing example pixels. Each of the example pixels was extracted from atraining set for which a labeled reference standard is available. Allpixels in the training set are thus assigned a label and a featurevector. A classifier can now establish a decision boundary in thefeature space which optimally separates the two classes (i.e.,foreground and background pixels).

A large number of sample pixels of red lesions, vasculature as well asbackground pixels are needed for training. A set of 40 fundus images,not used in the development or testing of any other part of the system,provided the training pixels. In all of the images the vasculature andred lesions were manually segmented. Each of these images waspreprocessed as described before training the pixel classificationprocedure. This training procedure needs to be completed only once butcan be performed multiple times; after it has finished, the system isready to process unseen images.

An important observation is the similarity, on a pixel level, in colorand response to first and second order Gaussian filters for the redlesions and the vasculature. To obtain the feature vectors, filteroutputs at each pixel location of Image_(PP) can be used. These filterscomprise all Gaussian derivatives up to and including second order atscales σ=1, 2, 4, 8, 16 pixels. Also the pixel intensities taken fromImage_(PP) were added to the feature vector. A kNN classifier producedthe best results with k=55. This method is not limited to the use of akNN classifier, any classifier known in the art can be used including,but not limited to a linear discriminant classifier, a quadraticdiscriminant classifier, support vector machines or boosted classifiers.

The kNN classifier (with k>1) allows for soft-classification in whicheach pixel is assigned a probability of it being a foreground pixel. Toaccomplish this, all k neighbors in the feature space of a query pixelcan be examined. When n neighbors were labeled as being a foregroundpixel, the posterior probability that the query pixel is a foregroundpixel itself p was determined byp=n/k  (2)

For the kNN classifier the optimized implementation as provided by Aryaet al., “An Optimal Algorithm for Approximate Nearest Neighbor Searchingin Fixed Dimensions,” Journal of the ACM, vol. 45, no. 6, pp. 891-923,1998 herein incorporated by reference in its entirety, can be used. FIG.6 a shows part of the green plane of a fundus image. Some red lesions601 a, b, c, d, are indicated. The probabilistic map of FIG. 6 a isshown in FIG. 6 d, with some red lesions 603 a, b, c, d indicated.

To extract possible candidate locations, the probabilistic map producedby the pixel classification can be thresholded. To determine thisprobability threshold, the maximum pixel value in the probability mapunder each of the objects marked in the ground truth can be examined.This can give an indication of the maximum amount of objects that can bedetected at a certain probability threshold. A good balance can beachieved between the number of detected spurious objects and the numberof detected red lesions with a probability threshold of 0.4. The methodis not limited to the use of a probability threshold at 0.4. Theprobability threshold can comprise values from 0 to 1. For example,probability threshold values can be 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6,0.7, 0.8, 0.9, and 1. After thresholding, a binary image containing anyred lesions and a large part of the vascular tree can be obtained.Because red lesions in general do not appear on larger (visible)vessels, they are disconnected from the vasculature. To obtain possiblecandidate locations, connected component analysis can be applied on thebinary objects. Any object which was too large to be a red lesion can beremoved. To determine a size threshold for this operation, a histogramcan be made of the sizes of all lesions in the reference standardsegmentation of the training set. A size threshold of 300 pixels wasfound to include 98% of all red lesions for images of the resolutionused. Most of the vasculature is connected, forming objects larger than300 pixels and can thus be removed by this step. The method is notlimited to the use of 300 pixels. For example, the method can comprisesize threshold values from 30 to 600, however the size threshold can beany size necessary, depending on the resolution of the image. Forexample, size threshold values can be 30, 50, 150, 200, 250, 300, 350,400, 450, 500, 550, 600, 650, 700, 750, and so on. What remains are anumber of small vessel fragments and those red lesions not connected tothe vasculature. In FIG. 7(a,b,c), a binary example image and theremaining connected components are shown. FIG. 7 a, depicts an exemplarybinary image. In FIG. 7 b, lesions 701 are shown after connectedcomponent analysis and removal of the large vasculature. In FIG. 7 c,lesions 702 are shown after a region growing procedure.

As with the mathematical morphology based approach the extracted objectsare not a good representation of the actual pathologies and objects inthe original image. Therefore, the same region growing technique asdescribed earlier can be applied to Image_(SC) to determine the actualshape of the objects.

The objects which are produced by the region growing procedure can thenbe post-processed such that any empty pixels in the grown objects arefilled. This post-processing can be done to facilitate featureextraction. The post-processed objects together form a set of candidateobjects.

c. Hybrid Candidate Object Detection

Both the pixel classification and the mathematical morphology basedsystem exhibit similar performance in terms of detected lesions.However, when the candidates of both systems are combined, a highernumber of true positive (TP) candidate objects can be detected. Thecombination can be done by putting all candidate objects in a set. Thenall objects can be checked to see if they overlap with another object inthe set. In case of overlap, one of the objects can be removed. Whenboth candidate detection systems detect the same lesion, it does notmatter which object is removed because the segmentation method used inboth approaches is the same. Overlap also often occurs when a regiongrowing seed point is placed in a normal area of the retina (e.g., dueto noise), the region growing procedure grows into neighboring redlesions and vessels. To exclude these relatively large spurious objects,the larger of two overlapping objects can be removed.

III. Candidate Classification System

The candidate detection step results in a set of candidate objects. Thepurpose of the candidate classification system is to classify each ofthese objects as either an abnormal object, which can be a red lesion,or a normal object, which can be a non-red lesion. To accomplish this,another classifier can be trained using example objects. These exampleobjects are first extracted from a training set using the candidateobject detection methods described earlier. Using the reference standardsegmentation of the training set, each of the example objects can beappropriately labeled. The kNN classifier showed the best performance.This method is not limited to the use of a kNN classifier; anyclassifier known in the art can be used including, but not limited to, alinear discriminant classifier, a quadratic discriminant classifier,support vector machines or techniques such as boosting.

Features

A total of thirteen different candidate object features were proposed byFrame et al., “A comparison of computer based classification methodsapplied to the detection of microaneurysms in ophthalmic fluoresceinangiograms,” Computers in Biology and Medicine, vol. 28, pp. 225-238,1998 herein incorporated by reference in its entirety, for use with theSpencer-Frame System. The features can be divided into two groups: shapefeatures (1-4) and pixel intensity features (5-13). With just thesethirteen features, system performance is limited.

Several new features are disclosed to improve performance. To determinewhat features can be added, the behavior of the system using theSpencer-Frame feature set was observed. These showed a significantnumber of FP objects on the vasculature. To counter this, Gaussianfilterbank outputs (Feature 19) were added to the feature set as well asa feature which can determine if an object is on an elongated structure(Feature 20). As fluorescein angiograms do not contain color, colorfeatures were not used in the Spencer-Frame system. Since the color ofthe images is an extra source of information, features 15-18 were added.Feature 14 eliminates elongated structures. Feature 21 can improve thedetection of larger circular objects. The feature set is listed below.Some features can be obtained at several image scales σ. The method isnot limited to the feature list below. For example, larger scales can beused for images with higher image resolutions.

1. The area a=Σ_(jεΩ)1 where Ω is the set of pixels in the candidate.

2. The perimeter p which is approximated using the chaincodes, see H.Freeman, “On Encoding Arbitrary Geometric Configurations,” IRETransactions on Electronic Computers, vol. 10, pp. 260-268, 1961 hereinincorporated by reference in its entirety, of the object. The number ofodd n_(o) and even n_(e) chaincodes are counted then p=n_(o)√{squareroot over (2)}+n_(e).

3. The aspect ratio r=l/w. l is the length of the largest and w of thesecond largest eigenvector of the covariance matrix of the object.

4. The circularity $c = {\frac{p^{2}}{4\pi\quad a}.}$

5. The total intensity of the object in the original green plane imagei_(green)=Σ_(jεΩ)g_(j) where g_(j) represents the j-th pixel in theoriginal green plane image.

6. The total intensity of the object in the shade corrected imagei_(sc)=Σ_(jεΩ)s_(j) where s_(j) represents the j-th pixel in the shadecorrected image.

7. The mean intensity under the object in the original green plane imagem_(green)=i_(green)/a.

8. The mean intensity under the object in the shade corrected imagem_(sc)=i_(sc)/a.

9. The normalized intensity in the original green plane image${NI}_{green} = {\frac{1}{\sigma}\left( {i_{green} - x} \right)}$where σ and x are the standard deviation and average pixel value ofI_(bg).

10. The normalized intensity in the shade corrected image${NI}_{sc} = {\frac{1}{\sigma}i}$

11. The normalized mean intensity in the original image${NM}_{green} = {\frac{1}{\sigma}{\left( {m_{green} - x} \right).}}$

12. The normalized mean intensity in the shade corrected image${NM}_{sc} = {\frac{1}{\sigma}m_{green}}$

13. The intensity of the region growing seed in the match filtered imageI_(match).

14. The compactness$v = \sqrt{\frac{{\sum\limits_{j = 1}^{n}d_{j}} - \overset{\_}{d}}{n}}$where d_(j) is the distance from the centroid of the object to its j-thboundary pixel and d is the mean of all the distances from the centroidto all the edge pixels. Here n is the number of edge pixels.

15. The difference between the mean pixel values inside the object andmean values in a circular region centered on the object in the red planefrom the RGB color space${{rd}(j)} = {{\frac{1}{n_{\Omega}}\Sigma_{j\quad ɛ\quad\Omega}R_{j}} - {\frac{1}{n_{\Theta}}\Sigma_{j\quad ɛ\quad\Theta}{R_{j}.}}}$Here Ω is the set of pixels in the candidate and Θ is the set of pixelsincluded in the circular region excluding the pixels in Ω. R_(j) is thej-th pixel in the red image plane. The diameter of the circular regionis determined by taking the maximum distance from the object centroid toany of the edges of an axis aligned bounding box and adding three. n_(Ω)and n_(Θ) are the number of pixels in Ω and Θ respectively.

16. As feature 15 but extracted from the green image plane.

17. As feature 15 but extracted from the blue image plane.

18. As feature 15 but extracted from the hue image plane taken from theHSI color space.

19. The mean and standard deviation of filter outputs under the object.The filters consist of the Gaussian and its derivatives up to secondorder at scales σ=1,2,4,8 pixels. The total amount of features obtainedin this way is 2(4×6)=48.

20. The average value under the object of the absolute difference of thetwo largest eigenvalues of the Hessian tensor. The scale σ=2 is used forthe Gaussian partial derivatives that make up the Hessian.

21. The average output under the object of an iris filter as describedin H. Kobatake and Y. Yoshinaga, “Detection of Spicules on MammogramBased on Skeleton Analysis,” IEEE Transactions on Medical Imaging, vol.15, no. 3, pp. 235-245, 1996 herein incorporated by reference in itsentirety, used on I_(sc) with a minimum circle radius of 4, maximumradius of 12 and 8 directions.

22. The Determinant and eigenvalues of the Hessian matrix for eachpixel.

In some cases, the use of feature selection schemes can help improve theperformance of a classifier. Several feature selection methods can beused on the feature set defined above. The method is not limited to theuse of a feature selection scheme; the method can use any or all of thefeatures disclosed.

IV. Examples

Method

Referring to FIG. 1, an exemplary method 100 is shown. The first step isan image pre-processing step 200, followed by a candidate detection step300, and finally a candidate classification step 400. Referring to FIG.2, the image pre-processing step 200, converts an Image_(ORG) 201 intoan Image_(SC) 203 by reducing intensity variation at 202.

Referring to FIGS. 3 a, b, and c, the candidate detection step 300, canbe performed in three ways. The candidate detection step 300 can beperformed as a Mathematical Morphology based step as in FIG. 3 a (300a), as a Pixel Classification based step as in FIG. 3 b (300 b), or ahybrid of Pixel Classification and Mathematical Morphology as in FIG. 3c (300 c).

Referring to FIG. 3 a, a morphological transformation is applied to theImage_(SC) at block 301, producing a Vasculature Map. The VasculatureMap is subtracted from the Image_(SC) at block 302, to produce anImage_(LESION). A matched filter is applied to the Image_(LESION) atblock 303, producing an Image_(MATCH). A brightness threshold is appliedto the Image_(MATCH) at block 304, producing a binary image,Image_(BIN-MM). A set of Seed Coordinates 306 is generated from theImage_(BIN-MM) at block 305. A region-growing procedure using the SeedCoordinates is applied to Image_(SC) at block 306 to generate a FinalSet of Candidate Objects.

Referring to FIG. 3 b, an Untrained Classifier is trained based off of aTraining Set, producing a Trained Classifier at block 307. Brightlesions are removed from the Image_(SC) at block 308, producing anImage_(PP). Pixel feature vectors are obtained from the Image_(PP) atblock 309. The Trained Classifer classifies the pixels of the Image_(PP)as foreground pixels or background pixels at block 310, producing aprobability map at block 311, Image_(PROB), of posterior probabilitiesof a pixel being a foreground or a background pixel. At block 312 theImage_(PROB) is thresholded to produce a binary image, Image_(BIN-PC).Large objects are removed from Image_(BIN-PC) at block 313 and a set ofSeed Coordinates is generated from the Image_(BIN-PC) at block 314. Aregion-growing procedure using the Seed Coordinates is applied toImage_(SC) at block 315 to generate a Final Set of Candidate Objects.

Referring to FIG. 3 c, the mathematical morphology method of 300 a canbe performed as described above except the region-growing procedureusing the Seed Coordinates as applied to Image_(SC) generates a FirstSet of Candidate Objects. The pixel classification method of 300 b canbe performed as described above except that the region-growing procedureusing the Seed Coordinates as applied to Image_(SC) generates a SecondSet of Candidate Objects. The First Set of Candidate Objects and theSecond Set of Candidate Objects can be combined at block 318. Thecombined Sets of Candidate objects form a Final Set of CandidateObjects. Methods 300 a and 300 b can be performed concurrently or in anyorder in the same or different computing environments.

Referring to FIG. 4, candidate classification 400, an UntrainedClassifier can be trained based off of a Training Set at block 401,producing a Trained Classifier. Each object in the Set of CandidateObjects can be compared to the Set of Features by the Trained Classifierat block 402 and each object can be classified as an abnormal objectwith a certain probability or a normal object with a certain probabilityat block 403.

Exemplary Computing System

FIG. 10 is a block diagram illustrating an exemplary operatingenvironment for performing the disclosed method. This exemplaryoperating environment is only an example of an operating environment andis not intended to suggest any limitation as to the scope of use orfunctionality of operating environment architecture. Neither should theoperating environment be interpreted as having any dependency orrequirement relating to any one or combination of components illustratedin the exemplary operating environment.

The method can be operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the system and methodinclude, but are not limited to, personal computers, server computers,laptop devices, and multiprocessor systems. Additional examples includeset top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The method may be described in the general context of computerinstructions, such as program modules, being executed by a computer.Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types. The system and method may alsobe practiced in distributed computing environments where tasks areperformed by remote processing devices that are linked through acommunications network. In a distributed computing environment, programmodules may be located in both local and remote computer storage mediaincluding memory storage devices.

The method disclosed herein can be implemented via a general-purposecomputing device in the form of a computer 1001. The components of thecomputer 1001 can include, but are not limited to, one or moreprocessors or processing units 1003, a system memory 1012, and a systembus 1013 that couples various system components including the processor1003 to the system memory 1012.

The system bus 1013 represents one or more of several possible types ofbus structures, including a memory bus or memory controller, aperipheral bus, an accelerated graphics port, and a processor or localbus using any of a variety of bus architectures. By way of example, sucharchitectures can include an Industry Standard Architecture (ISA) bus, aMicro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, aVideo Electronics Standards Association (VESA) local bus, and aPeripheral Component Interconnects (PCI) bus also known as a Mezzaninebus. This bus, and all buses specified in this description can also beimplemented over a wired or wireless network connection. The bus 1013,and all buses specified in this description can also be implemented overa wired or wireless network connection and each of the subsystems,including the processor 1003, a mass storage device 1004, an operatingsystem 1005, image analysis software 1006, digital fundus image data1007, a network adapter 1008, system memory 1012, an Input/OutputInterface 1010, a display adapter 1009, a display device 1011, a humanmachine interface 1002, and a digital image capture device 1016, can becontained within one or more remote computing devices 1014 a,b,c atphysically separate locations, connected through buses of this form, ineffect implementing a fully distributed system.

The computer 1001 typically includes a variety of computer readablemedia. Such media can be any available media that is accessible by thecomputer 1001 and includes both volatile and non-volatile media,removable and non-removable media. The system memory 1012 includescomputer readable media in the form of volatile memory, such as randomaccess memory (RAM), and/or non-volatile memory, such as read onlymemory (ROM). The system memory 1012 typically contains data such asdigital fundus image data 1007 and/or program modules such as operatingsystem 1005 and image analysis software 1006 that are immediatelyaccessible to and/or are presently operated on by the processing unit1003.

The computer 1001 may also include other removable/non-removable,volatile/non-volatile computer storage media. By way of example, FIG. 10illustrates a mass storage device 1004 which can provide non-volatilestorage of computer code, computer readable instructions, datastructures, program modules, and other data for the computer 1001. Forexample, a mass storage device 1004 can be a hard disk, a removablemagnetic disk, a removable optical disk, magnetic cassettes or othermagnetic storage devices, flash memory cards, CD-ROM, digital versatiledisks (DVD) or other optical storage, random access memories (RAM), readonly memories (ROM), electrically erasable programmable read-only memory(EEPROM), and the like.

Any number of program modules can be stored on the mass storage device1004, including by way of example, an operating system 1005 and imageanalysis software 1006. Each of the operating system 1005 andapplication software 1006 (or some combination thereof) may includeelements of the programming and the application software 1006. Digitalfundus image data 1007 can also be stored on the mass storage device1004. Digital fundus image data 1007 can be stored in any of one or moredatabases known in the art. Examples of such databases include, DB2®,Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL,and the like. The databases can be centralized or distributed acrossmultiple systems.

A user can enter commands and information into the computer 1001 via aninput device (not shown). Examples of such input devices include, butare not limited to, a keyboard, pointing device (e.g., a “mouse”), amicrophone, a joystick, a serial port, a scanner, and the like. Theseand other input devices can be connected to the processing unit 1003 viaa human machine interface 1002 that is coupled to the system bus 1013,but may be connected by other interface and bus structures, such as aparallel port, game port, or a universal serial bus (USB).

Digital fundus image data 1007 can be obtained by using a digital imagecapture device 1016 to obtain a digital image of a subject fundus 1017.A digital image capture device 1016 can include, but is not limited to,a digital still image camera, a video capture card, a digital videocamera, and the like. The digital fundus image data 1007 generated by adigital image capture device 1016 can enter the computer 1001 via theinput/output interface 1010. The digital fundus image data 1007 can bestored in mass storage device 1004 and transferred to system memory 1012as digital fundus image data 1007 to be used in image analysis software1006. Digital fundus image data 1007 can also be obtained from otherstorage sources and transferred into the computer 1001 via theinput/output interface or removable computer readable media.

A display device 1011 can also be connected to the system bus 1013 viaan interface, such as a display adapter 1009. A computer 1001 can havemore than one display adapter 1009 and a computer 1001 can have morethan one display device 1011. For example, a display device can be amonitor, an LCD (Liquid Crystal Display), or a projector. In addition tothe display device 1011, other output peripheral devices can includecomponents such as speakers (not shown) and a printer (not shown) whichcan be connected to the computer 1001 via Input/Output Interface 1010.

The computer 1001 can operate in a networked environment using logicalconnections to one or more remote computing devices 1014 a,b,c. By wayof example, a remote computing device can be a personal computer,portable computer, a server, a router, a network computer, a peer deviceor other common network node, and so on. Logical connections between thecomputer 1001 and a remote computing device 1014 a,b,c can be made via alocal area network (LAN) and a general wide area network (WAN). Suchnetwork connections can be through a network adapter 1008. A networkadapter 1008 can be implemented in both wired and wireless environments.Such networking environments are commonplace in offices, enterprise-widecomputer networks, intranets, and the Internet 1015.

For purposes of illustration, application programs and other executableprogram components such as the operating system 1005 are illustratedherein as discrete blocks, although it is recognized that such programsand components reside at various times in different storage componentsof the computing device 1001, and are executed by the data processor(s)of the computer. An implementation of image analysis software 1006 maybe stored on or transmitted across some form of computer readable media.Computer readable media can be any available media that can be accessedby a computer. By way of example, and not limitation, computer readablemedia may comprise “computer storage media” and “communications media.”“Computer storage media” include volatile and non-volatile, removableand non-removable media implemented in any method or technology forstorage of information such as computer readable instructions, datastructures, program modules, or other data. Computer storage mediaincludes, but is not limited to, RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed by acomputer.

The processing of the disclosed method can be performed by softwarecomponents. The disclosed method may be described in the general contextof computer-executable instructions, such as program modules, beingexecuted by one or more computers or other devices. Generally, programmodules include computer code, routines, programs, objects, components,data structures, etc. that perform particular tasks or implementparticular abstract data types. The disclosed method may also bepracticed in grid-based and distributed computing environments wheretasks are performed by remote processing devices that are linked througha communications network. In a distributed computing environment,program modules may be located in both local and remote computer storagemedia including memory storage devices

Implementation

A set of 40 images (Set 1) was used to train a kNN classifier in thefirst candidate object extraction step. A second set of 100 images (Set2) was used to train and test the complete system. All of the images inboth sets were acquired digitally.

A. Set 1

Set 1 is a publicly available database of 40 manually segmented fundusimages. All the vasculature has been manually segmented. All red lesionswere manually segmented and added to the existing segmentation.

B. Set 2

Set 2 contains a total of 100 images and was used to train and test thepresented system. Because screening data typically contain less than 10%pathological images, Set 2 was not assembled using just screeningimages. Of the images in Set 2, 74 were obtained at a tertiary referralhospital. A team of two ophthalmologists (MDA and MS), both retinaldisease specialists, selected 37 images with and 37 images withoutpathology. These images were captured using a Topcon® TRC-50 at 45°field of view. The remaining 26 images were taken from a DR screeningprogram in the Netherlands. These images were captured using a Canon®CR5 non-mydriatic 3CCD camera at 45° field of view. Of these images 13contained red lesions, and 13 showed no visible red lesions. Somepatients have two images included in the set, but these were never ofthe same eye.

All images were JPEG compressed. This lossy image compression format isnot ideal for image processing but is commonly used in a screeningsetting. The size of the images is 768×576 pixels and the field of viewis approximately 540 pixels in diameter. All personally identifiableinformation was stripped from all the images so that only the raw imagedata was available for this research.

The team separately annotated the entire set of 100 images. Theannotation was performed by manually marking all pixels of a red lesionfor every possible red lesion in the set. All 100 images were annotatedin random order. The annotation of the first ophthalmologist was takenas the reference standard. According to the reference standard, a totalof 55 images from the entire set contained pathologies. In these 55images, a total of 858 red lesions were identified. Next, the completeset was randomly split into a test and a training set of 50 images each.Both the test and the training set contained 12 screening and 37clinical images. As for red lesions, the test set contained 27 imageswith a total of 348 lesions, while the training set contained 28 imageswith a total of 510 red lesions. If two images were obtained fromdifferent eyes of the same patient, both images were assigned to eitherthe test or training set. Also, the clinical and screening images weredistributed equally over both sets.

A number of different system setups were tested to determine whichprovides the best performance:

-   -   The hybrid system (HS) using both the hybrid object candidate        extraction system and the complete set of features as defined        earlier.    -   The pixel classification system (PC) using the pixel        classification based candidate extraction system and the        complete set of features except feature 13 which is unique for        the mathematical morphology based system.    -   The mathematical morphology based system using the complete set        of features (MMA).    -   The mathematical morphology based system using the original 13        features as proposed by Spencer-Frame (MMO).

All systems used a kNN classifier for the classification of thecandidate objects. Set 2 was also annotated by a second ophthalmologist.This allowed a comparison of the performance of the automatic systemswith that of a human expert.

A. Settings

For each system, the parameter settings as detailed in the previoussections were used.

For the candidate object classification step, the optimal value of k forthe kNN classifier was determined using the training set. For allsystems, an optimal value of k=11 was found except for the MMA wherek=9.

B. Results

Table I shows an overview of the extraction performance of all testedsystems when detecting candidate red lesion objects in the test set.TABLE I Hybrid Pixel Mathematical Human System Classification MorphologyExpert Total nr. 14906 13680 10209 478 of candidates Lesions 33 (9.48%)49 (14.1%) 49 (14.1%) 108 (31.0%) missed

Evaluating the final classification performance of a red lesiondetection system is application dependent. When using the method as apre-screening system as described earlier, its purpose can be todetermine whether or not an image is normal, i.e., free of red lesions,or abnormal, i.e. containing one or more red lesions. Receiver OperatingCharacteristic (ROC) curve analysis can be used to evaluate this type ofperformance. The ROC curve plots the sensitivity against the specificityof a system. It is an indication of the capability of the system todiscriminate between those images containing red lesions and those thatdo not contain red lesions. FIG. 8(a) shows the ROC curves of the HS andthe MMO. Because the human expert provides one sensitivity/specificitypair its performance can be represented by a single point in FIG. 8 a,b.

As a tool to assist a human expert in detecting subtle red lesions,performance can be measured using Free-Response Operating Characteristic(FROC) analysis. When the aforementioned method is used for assisting ahuman, the goal of the method is to detect as many lesions as possiblewhile not generating an excessive number of FPs. FIG. 8(b) shows theFROC curves of both the HS and the MMO. An FROC curve plots thesensitivity of the system with regard to all the red lesions in the testset against the average number of FPs per image. This gives anindication of the actual red lesion detection performance of thesystems. The horizontal axis in FIG. 8(b) has a logarithmic scale.

Table II shows the performance of all systems at two important operatingpoints. The first operating point has the same sensitivity per image asthe human expert, 0.89, and the second operating point is the point withthe highest specificity at which the sensitivity is 1. TABLE IISensitivity Specificity Sensitivity Average per per per nr. of Systemimage image lesion FP per image HS 0.89 0.96 0.26 0.44 1.00 0.87 0.310.80 PC 0.89 0.96 0.21 0.36 1.00 0.53 0.46 4.10 MMA 0.89 0.96 0.22 0.421.00 0.60 0.42 2.54 MMO 0.89 0.48 0.34 4.38 1.00 0.00 0.86 1.96 HumanExpert 0.89 1.00 0.68 2.60

The results show that the hybrid system described herein, when used as apre-screening tool, can operate at a sensitivity of 100% combined with87% specificity. This means a large decrease in images which have to bescreened by an ophthalmologist. The performance of the system on a perimage basis is close to that of a human expert. Using previous systemsin a pre-screening setup is not practical because it is not able toattain 100% sensitivity without its specificity dropping to 0%.

The first candidate detection step of the hybrid system combined thedetected candidates of the mathematical morphology with the candidatesof the pixel classification based system. It was important to miss asfew red lesions as possible in the first candidate selection stepbecause any lesions lost in this step could not be retrieved later.Therefore, each system can be adjusted for maximum sensitivity in thefirst step, assuring that the largest number of lesions is detected.This causes a large number of spurious candidate objects to be detected.Table I shows that the hybrid system detected more red lesions than theother systems, but it also detected the largest number of candidateobjects overall. There are a number of reasons why 33 lesions weremissed by the hybrid detection system. Approximately half of them arevery close or connected with the vasculature; both the mathematicalmorphology as well as the pixel classification based approach hadproblems finding these. An example of this type of missed lesion isshown in FIG. 9. The other half of the missed lesions were extremelysubtle or blurred. Some of these objects were not detected at all, andif they were detected, the region growing procedure often failed to findthe object border and grew into nearby objects. FIG. 9 shows an exampleof a final result of the hybrid red lesion detection system. Truepositives (TPs) 901 of the automatic system, false positives (FPs) 902of the automatic system, and false negatives (FNs) 903 of the automaticsystem are indicated. Objects 904 missed in the candidate detection stepare indicated. FPs 905 and FNs 906 of the human expert are alsoindicated. The system was operating at a per imagesensitivity/specificity of 1.00/0.87 and a per lesion sensitivity of0.31 at 0.80 average FPs 902 per image. This image contains 14 redlesions according to the reference standard. The second observercorrectly identified 10 of those (71.4%), produced 6 FPs 902 and 4 FNs903. The automatic system correctly identified 6 red lesions (42.9%),produced 3 FPs 902 and 8 FNs 903. One of the FNs 903 is a lesion thatwas missed in the first hybrid candidate detection step and thus thesystem will not be able to detect this lesion regardless of theoperating point that is used. All detected objects have been marked onthe image using different shapes. When an object belongs to multipleclasses (e.g. is both a TP 901, of the hybrid system and a FN 906 of thehuman expert) the indicator of the second class is displayed smallerclose to the first class. The contrast in this image has been adjustedfor optimal display.

The human expert was asked to mark any possible red lesions found in theset; this resulted in 478 marked objects and a total of 108 missed. Sogiven the task of detecting possible red lesions, the human expert wasless sensitive but much more specific than the HS. The HS was operatingat 3.80 average FPs per image; it detected 27 red lesions which were notfound by the human expert. On average this is one lesion per positiveimage in the test set. When the operating point was changed so that theaverage number of FP per image was increased to 9.94, the system found41 lesions the human expert did not detect. This is a good indicationthat the HS could be useful as a second reader that would help anophthalmologist detect subtle lesions.

As FIG. 8(a), 8(b) and Table II show, the human expert did not detectevery abnormal image in the test set nor found all red lesions. Thisdisagreement with the reference standard has several possible reasons.Because the images have been JPEG compressed and many red lesions arevery small, it is easy for some to be missed. Further, it is possiblethat the second expert read the images differently than the first. Whenan ophthalmologist performs a clinical reading of a color fundusphotograph, the precise number of lesions is less important than theirlocation. Observation of a number of cases showed that a significantnumber of FN lesions of the second observer were located outside of thevascular arch. Lesions in that part of the fundus are clinically lessrelevant. It can be that the second human expert interpreted the imagesmore in terms of clinical relevance of the lesions than the expert whodefined the reference standard even though both were given the same taskdescription. This effect is visible in FIG. 9. Most of the FN of thehuman expert were located outside of the vascular arch while most TP andFP were located inside of the vascular arch.

While the human expert missed 31% of all lesions, its per lesionsensitivity, 68%, was significantly higher than any of the automaticsystems. This was caused by the fact that although more lesions weremissed by the human expert, the number of detected lesions which wereclassified correctly was higher. The hybrid system reached 100%sensitivity on a per image basis when the sensitivity on a per lesionbasis was at 30%. The probable reason for this sensitivity difference isthat some of the images contained more red lesions than others, andthus, the system did not need a very high per lesion sensitivity as longas it was able to detect at least one of the lesions in each of theimages. On average, the 27 positive images in the test set contained12.9 lesions with a standard deviation of 9.47. The high standarddeviation indicated the diversity of the test set, in which both imagescontaining a large number of lesions (5 images with more than 20 redlesions) and subtle images (9 images with less than 7 lesions) wererepresented. Generally the more lesions an image contained, the easierit was for the automatic system to detect it as a positive. The test setused partly consisted of images obtained in a tertiary referral hospitalin addition to images obtained in a screening program. There was a smalldifference in performance in favor of the screening images

Through the use of advanced pattern recognition techniques, some partsof the system can be optimized further. This includes the use of moreadvanced classifiers such as support vector machines or techniques suchas boosting. By increasing the amount of training data for the candidateobject classification, overall system performance can be furtherimproved. Large red lesions are not always completely segmented by theused region growing procedure; the region growing procedure can bereplaced by a different segmentation method.

The total time required by the fully automatic hybrid system to processa single image was approximately 15 minutes. All methods were conductedusing an Intel Pentium IV based computer running at 1.7 GHz. Most ofthis time was spent during the initial candidate detection step, i.e.,the pixel classification procedure. However, this was just oneimplementation and performance can be improved.

Summarizing, the system and method presented detected every positiveimage in the test set while classifying only a small number of negativeimages as positive.

The foregoing detailed description has been given for understandingexemplary implementations of the invention only, and no unnecessarylimitations should be understood therefrom as modifications will beobvious to those skilled in the art without departing from the scope ofthe appended claims and their equivalents.

1. A method of identifying a subject with a disease of the retinacomprising automatically detecting in a digital color fundus photographof the subject abnormal objects in an Image_(ORG), comprising: a.reducing intensity variation in the Image_(ORG) producing ashade-corrected image, Image_(SC) and a background image, Image_(BG); b.automatically detecting candidate objects in the Image_(SC); and c.automatically classifying the candidate objects as abnormal or normal,wherein classification of an object as abnormal identifies a subjectwith a disease of the retina.
 2. The method of claim 1, whereinautomatically detecting candidate objects comprises: a. applying amorphological transformation to the Image_(SC) to discriminate circularfrom elongated regions producing a vasculature map; b. subtracting thevasculature map from the Image_(SC) to produce an Image_(LESION); c.applying a matched filter to the Image_(LESION) to enhance the contrastbetween background regions and abnormal regions producing anImage_(MATCH); d. applying a brightness threshold to the Image_(MATCH)to produce a binary digital fundus photograph, Image_(BIN-MM), whereinthe threshold is above the modal value of the Image_(MATCH), generatinga plurality of one or more seed coordinates; and e. applying aregion-growing procedure using the plurality of one or more seedcoordinates to the Image_(SC) to grow a set of candidate objects.
 3. Themethod of claim 2, wherein the brightness threshold is from about 2 toabout
 6. 4. The method of claim 3, wherein the brightness threshold isabout
 4. 5. The method of claim 2, wherein the set of candidate objectscomprises image coordinates.
 6. The method of claim 2, wherein the setof candidate objects comprises an image.
 7. The method of claim 1,wherein automatically detecting candidate objects comprises: a. removingbright lesions in the Image_(SC) to produce an Image_(PP); b. training aclassifier with a supervised procedure using example pixels extractedfrom a labeled reference standard training set; c. obtaining pixelfeature vectors from the Image_(PP) comprising applying filters fordetermining pixel intensities or other characteristics of pixels at eachpixel location or pixel region in the Image_(PP); d. automaticallyclassifying each of a plurality of the Image_(PP) pixels as foregroundor background pixels; e. assigning a posterior probability of being aforeground pixel to each Image_(PP) pixel according to the equationp=n/k, producing an Image_(PROB); f. removing the Image_(PROB) pixelswhere the posterior probability is less than a probability threshold,producing an Image_(BIN-PC); g. removing any object in theImage_(BIN-PC) with a size that is above a size threshold, generating aplurality of one or more seed coordinates; h. applying a region-growingprocedure using the plurality of one or more seed coordinates to theImage_(SC) to grow a set of candidate objects; and i. filling emptypixels within the grown objects of the set of candidate objects.
 8. Themethod of claim 7, wherein the classifier is selected from the groupconsisting of: a. a k-Nearest Neighbors classifier; b. a lineardiscriminant classifier; c. a quadratic discriminant classifier; and d.a support vector machine.
 9. The method of claim 7, wherein theprobability threshold is from about 0 to about
 1. 10. The method ofclaim 9, wherein the probability threshold is about 0.4.
 11. The methodof claim 7, wherein automatically classifying each of a plurality of theImage_(PP) pixels as foreground or background pixels comprises asupervised procedure.
 12. The method of claim 7, wherein the sizethreshold is from about 30 pixels to about 600 pixels.
 13. The method ofclaim 12, wherein the size threshold is about 300 pixels.
 14. The methodof claim 7, wherein the set of candidate objects comprises imagecoordinates.
 15. The method of claim 7, wherein the set of candidateobjects comprises an image.
 16. The method of claim 1, whereinautomatically detecting candidate objects comprises: a. applyingmathematical morphology based candidate object detection to produce afirst set of candidate objects; b. applying a supervised pixelclassification based candidate object detection to produce. a second setof candidate objects; c. combining the first set of candidate objectsand the second set of candidate objects to form a third set of candidateobjects; d. identifying in the third set of candidate objects aplurality of pairs of overlapping objects; and e. removing one objectfrom each pair of overlapping objects in the third set of candidateobjects.
 17. The method of claim 1, wherein automatically detectingcandidate objects comprises: a. applying a morphological transformationto the Image_(SC) to discriminate circular from elongated regionsproducing a vasculature map; b. subtracting the vasculature map from theImage_(SC) to produce an Image_(LESION); c. applying a matched filter tothe Image_(LESION) to enhance the contrast between background regionsand abnormal regions producing an Image_(MATCH); d. applying abrightness threshold to the Image_(MATCH) to produce a binary digitalfundus photograph, Image_(BIN-MM), wherein the threshold is above themodal value of the Image_(MATCH), generating a first plurality of one ormore seed coordinates; e. applying a region-growing procedure using thefirst plurality of one or more seed coordinates to the Image_(SC) togrow a first set of candidate objects; f. removing bright lesions in theImage_(SC) to produce an Image_(PP); g. training a classifier with asupervised procedure using example pixels extracted from a labeledreference standard training set; h. obtaining pixel feature vectors fromthe Image_(PP) comprising applying filters for determining pixelintensities or other characteristics of pixels at each pixel location orpixel region in the Image_(PP); i. automatically classifying each of aplurality of the Image_(PP) pixels as foreground or background pixels;j. assigning a posterior probability of being a foreground pixel to eachImage_(PP) pixel according to the equation p=n/k, producing anImage_(PROB); k. removing the Image_(PROB) pixels where the posteriorprobability is less than a probability threshold, producing anImage_(BIN-PC); l. removing any object in the Image_(BIN-PC) with a sizethat is above a size threshold, generating a second plurality of one ormore seed coordinates; m. applying a region-growing procedure using thesecond plurality of one or more seed coordinates to the Image_(SC) togrow a second set of candidate objects; n. filling empty pixels in thegrown objects of the second set of candidate objects; o. combining thefirst set of candidate objects and the second set of candidate objectsto form a third set of candidate objects; p. identifying in the thirdset of candidate objects a plurality of pairs of overlapping objects;and q. removing one object from each pair of overlapping objects in thethird set of candidate objects.
 18. The method of claim 17, wherein theclassifier is selected from the group consisting of: a. a k-NearestNeighbors classifier; b. a linear discriminant classifier; c. aquadratic discriminant classifier; and d. a support vector machine. 19.The method of claim 17, wherein the brightness threshold is from about 2to about
 6. 20. The method of claim 19, wherein the brightness thresholdis about
 4. 21. The method of claim 17, wherein the probabilitythreshold is from about 0 to about
 1. 22. The method of claim 21,wherein the probability threshold is about 0.4.
 23. The method of claim17, wherein automatically classifying each of a plurality of theImage_(PP) pixels as foreground or background pixels comprises asupervised procedure.
 24. The method of claim 17, wherein the sizethreshold is from about 30 pixels to about 600 pixels.
 25. The method ofclaim 24, wherein the size threshold is about 300 pixels.
 26. The methodof claim 17, wherein the removed overlapping object is the larger objectof each pair.
 27. The method of claim 1, wherein automaticallyclassifying the candidate objects as abnormal or normal, whereinclassification of an object as abnormal identifies a subject with adisease of the retina comprises: a. training a classifier with asupervised procedure using example objects extracted from a labeledreference standard training set; b. comparing each of the candidateobjects using a set of features, wherein objects recognized by theclassifier as abnormal are classified as abnormal with a certainprobability and objects recognized by the classifier as normal areclassified as normal with a certain probability; and c. identifying asubject whose candidate objects are identified as abnormal with acertain probability as having a disease of the retina.
 28. The method ofclaim 27, wherein the classifier is selected from the group consistingof: a. a k-Nearest Neighbors classifier; b. a linear discriminantclassifier; c. a quadratic discriminant classifier; and d. a supportvector machine.
 29. The method of claim 27 wherein the set of featuresis selected from the group consisting of: a. the area a=Σ_(jεΩ)1 where Ωis the set of pixels in the candidate; b. the perimeter p which isapproximated using the chaincodes of the object; c. the number of oddn_(o) and even n_(e) chaincodes are counted then p=n_(o)√{square rootover (2)}+n_(e); d. the aspect ratio r=l/w, wherein l is the length ofthe largest and w of the second largest eigenvector of the covariancematrix of the object; e. the circularity${c = \frac{p^{2}}{4\pi\quad a}};$ f. the total intensity of the objectin the original green plane image i_(green)=Σ_(jεΩ)g_(j) where g_(j)represents the j-th pixel in the original green plane image; g. thetotal intensity of the object in the shade corrected imagei_(sc)=Σ_(jεΩ)s_(j) where s_(j) represents the j-th pixel in the shadecorrected image; h. the mean intensity under the object in the originalgreen plane image m_(green)=i_(green)/a; i. the mean intensity under theobject in the shade corrected image m_(sc)=i_(sc)/a; j. the normalizedintensity in the original green plane image${NI}_{green} = {\frac{1}{\sigma}\left( {i_{green} - x} \right)}$ whereσ and x are the standard deviation and average pixel value of I_(bg); k.the normalized intensity in the shade corrected image${{NI}_{sc} = {\frac{1}{\sigma}i}};$ l. the normalized mean intensity inthe original image${{NM}_{green} = {\frac{1}{\sigma}\left( {m_{green} - x} \right)}};$ m.the normalized mean intensity in the shade corrected image${{NM}_{sc} = {\frac{1}{\sigma}m_{green}}};$ n. the intensity of theregion growing seed in the match filtered image I_(match); o. thecompactness$v = \sqrt{\frac{{\sum\limits_{j = 1}^{n}d_{j}} - \overset{\_}{d}}{n}}$wherein d_(j) is the distance from the centroid of the object to itsj-th boundary pixel and d is the mean of all the distances from thecentroid to all the edge pixels and n is the number of edge pixels; p.the difference between the mean pixel values inside the object and meanvalues in a circular region centered on the object in the red plane fromthe RGB color space${{{rd}(j)} = {{\frac{1}{n_{\Omega}}{\sum\limits_{j\quad ɛ\quad\Omega}R_{j}}} - {\frac{1}{n_{\Theta}}{\sum\limits_{j\quad{ɛ\Theta}}R_{j}}}}},$wherein Ω is the set of pixels in the candidate and Θ is the set ofpixels included in the circular region excluding the pixels in Ω. R_(j)is the j-th pixel in the red image plane and the diameter of thecircular region is determined by taking the maximum distance from theobject centroid to any of the edges of an axis aligned bounding box andadding three. n_(Ω) and n_(Θ) are the number of pixels in Ω and Θrespectively; q. as feature p but extracted from the green image plane;r. as feature p but extracted from the blue image plane; s. as feature pbut extracted from the hue image plane taken from the HSI color space;t. the mean and standard deviation of filter outputs under the objectwherein the filters comprise the Gaussian and its derivatives up tosecond order at scales σ=1,2,4,8 pixels; u. the average value under theobject of the absolute difference of the two largest eigenvalues of theHessian tensor wherein the scale σ=2 is used for the Gaussian partialderivatives that make up the Hessian; v. the average output under theobject of an iris filter used on I_(SC) with a minimum circle radius of4, maximum radius of 12 and 8 directions; and w. adding the determinantand eigenvalues of the Hessian matrix of a pixel.
 30. The method ofclaim 1 wherein the objects classified as abnormal comprise:microaneurysms, dot hemorrhages, flame-shaped hemorrhages, sub-intimalhemorrhages, sub-retinal hemorrhages, pre-retinal hemorrhages,micro-infarctions, cotton-wool spots, and yellow exudates.
 31. A systemthat identifies a subject with a disease of the retina by detecting in adigital color fundus photograph of the subject abnormal objects in anImage_(ORG), comprising: a processor; and a computer readable mediumcoupled to said processor for storing a computer program comprising:computer code that reduces intensity variation in the Image_(ORG)producing a shade-corrected image, Image_(SC) and a background image,Image_(BG); computer code that automatically detects candidate objectsin the Image_(SC); and computer code that automatically classifies thecandidate objects as abnormal or normal, wherein classification of anobject as abnormal identifies a subject with a disease of the retina.32. The system of claim 31 wherein the computer code that automaticallydetects candidate objects in the Image_(SC) further comprises: computercode that implements mathematical morphology based candidate objectdetection; and computer code that implements supervised pixelclassification based candidate object detection.
 33. The system of claim31, wherein the computer code that automatically classifies thecandidate objects as abnormal or normal, wherein classification of anobject as abnormal identifies a subject with a disease of the retinafurther comprises: computer code that trains a classifier with asupervised procedure using example objects extracted from a labeledreference standard training set; computer code that compares each of thecandidate objects using a set of features, wherein objects recognized bythe classifier as abnormal are classified as abnormal with a certainprobability and objects recognized by the classifier as normal areclassified as normal with a certain probability; and computer code thatidentifies a subject whose candidate objects are identified as abnormalwith a certain probability as having a disease of the retina.
 34. Asystem that identifies a subject with a disease of the retina bydetecting in a digital color fundus photograph of the subject abnormalobjects in an Image_(ORG), comprising: a storage device for storingdigital fundus image data; and a processor operably coupled to thestorage device for performing the steps of: reducing intensity variationin the Image_(ORG) producing a shade-corrected image, Image_(SC) and abackground image, Image_(BG); automatically detecting candidate objectsin the Image_(SC); and automatically classifying the candidate objectsas abnormal or normal, wherein classification of an object as abnormalidentifies a subject with a disease of the retina.
 35. The system ofclaim 34 wherein the automatically detecting candidate objects in theImage_(SC) further comprises: applying mathematical morphology basedcandidate object detection; and applying supervised pixel classificationbased candidate object detection.
 36. The system of claim 34, whereinautomatically classifying the candidate objects as abnormal or normal,wherein classification of an object as abnormal identifies a subjectwith a disease of the retina further comprises: training a classifierwith a supervised procedure using example objects extracted from alabeled reference standard training set; comparing each of the candidateobjects using a set of features, wherein objects recognized by theclassifier as abnormal are classified as abnormal with a certainprobability and objects recognized by the classifier as normal areclassified as normal with a certain probability; and identifying asubject whose candidate objects are identified as abnormal with acertain probability as having a disease of the retina.